{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Keras for Text Classification\n",
    "\n",
    "## Learning Objectives\n",
    "1. Learn how to create a text classification datasets using BigQuery.\n",
    "1. Learn how to tokenize and integerize a corpus of text for training in Keras.\n",
    "1. Learn how to do one-hot-encodings in Keras.\n",
    "1. Learn how to use embedding layers to represent words in Keras.\n",
    "1. Learn about the bag-of-word representation for sentences.\n",
    "1. Learn how to use DNN/CNN/RNN model to classify text in keras.\n",
    "\n",
    "## Introduction\n",
    "\n",
    "\n",
    "In this notebook, we will implement text models to recognize the probable source (Github, Tech-Crunch, or The New-York Times) of the titles we have in the title dataset we constructed in the first task of the lab.\n",
    "\n",
    "In the next step, we will load and pre-process the texts and labels so that they are suitable to be fed to a Keras model. For the texts of the titles we will learn how to split them into a list of tokens, and then how to map each token to an integer using the Keras Tokenizer class. What will be fed to our Keras models will be batches of padded list of integers representing the text. For the labels, we will learn how to one-hot-encode each of the 3 classes into a 3 dimensional basis vector.\n",
    "\n",
    "Then we will explore a few possible models to do the title classification. All models will be fed padded list of integers, and all models will start with a Keras Embedding layer that transforms the integer representing the words into dense vectors.\n",
    "\n",
    "The first model will be a simple bag-of-word DNN model that averages up the word vectors and feeds the tensor that results to further dense layers. Doing so means that we forget the word order (and hence that we consider sentences as a “bag-of-words”). In the second and in the third model we will keep the information about the word order using a simple RNN and a simple CNN allowing us to achieve the same performance as with the DNN model but in much fewer epochs.\n",
    "\n",
    "Each learning objective will correspond to a __#TODO__ in the [student lab notebook](../labs/keras_for_text_classification.ipynb) -- try to complete that notebook first before reviewing this solution notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "from google.cloud import bigquery\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext google.cloud.bigquery"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Replace the variable values in the cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PROJECT = \"cloud-training-demos\"  # Replace with your PROJECT\n",
    "BUCKET = PROJECT  # defaults to PROJECT\n",
    "REGION = \"us-central1\"  # Replace with your REGION\n",
    "SEED = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a Dataset from BigQuery \n",
    "\n",
    "Hacker news headlines are available as a BigQuery public dataset. The [dataset](https://console.cloud.google.com/bigquery?project=bigquery-public-data&page=table&t=full&d=hacker_news&p=bigquery-public-data&redirect_from_classic=true) contains all headlines from the sites inception in October 2006 until October 2015. \n",
    "\n",
    "Here is a sample of the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bigquery --project $PROJECT\n",
    "\n",
    "SELECT\n",
    "    url, title, score\n",
    "FROM\n",
    "    `bigquery-public-data.hacker_news.full`\n",
    "WHERE\n",
    "    LENGTH(title) > 10\n",
    "    AND score > 10\n",
    "    AND LENGTH(url) > 0\n",
    "LIMIT 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's do some regular expression parsing in BigQuery to get the source of the newspaper article from the URL. For example, if the url is http://mobile.nytimes.com/...., I want to be left with <i>nytimes</i>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bigquery --project $PROJECT\n",
    "\n",
    "SELECT\n",
    "    ARRAY_REVERSE(SPLIT(REGEXP_EXTRACT(url, '.*://(.[^/]+)/'), '.'))[safe_offset (1)] AS source,\n",
    "    COUNT(title) AS num_articles\n",
    "FROM\n",
    "    `bigquery-public-data.hacker_news.full`\n",
    "WHERE\n",
    "    REGEXP_CONTAINS(REGEXP_EXTRACT(url, '.*://(.[^/]+)/'), '.com$')\n",
    "    AND LENGTH(title) > 10\n",
    "GROUP BY\n",
    "    source\n",
    "ORDER BY num_articles DESC\n",
    "  LIMIT 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have good parsing of the URL to get the source, let's put together a dataset of source and titles. This will be our labeled dataset for machine learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "regex = '.*://(.[^/]+)/'\n",
    "\n",
    "\n",
    "sub_query = \"\"\"\n",
    "SELECT\n",
    "    title,\n",
    "    ARRAY_REVERSE(SPLIT(REGEXP_EXTRACT(url, '{0}'), '.'))[safe_offset (1)] AS source\n",
    "    \n",
    "FROM\n",
    "    `bigquery-public-data.hacker_news.full`\n",
    "WHERE\n",
    "    REGEXP_CONTAINS(REGEXP_EXTRACT(url, '{0}'), '.com$')\n",
    "    AND LENGTH(title) > 10\n",
    "\"\"\".format(regex)\n",
    "\n",
    "\n",
    "query = \"\"\"\n",
    "SELECT \n",
    "    LOWER(REGEXP_REPLACE(title, '[^a-zA-Z0-9 $.-]', ' ')) AS title,\n",
    "    source\n",
    "FROM\n",
    "  ({sub_query})\n",
    "WHERE (source = 'github' OR source = 'nytimes' OR source = 'techcrunch')\n",
    "\"\"\".format(sub_query=sub_query)\n",
    "\n",
    "print(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For ML training, we usually need to split our dataset into training and evaluation datasets (and perhaps an independent test dataset if we are going to do model or feature selection based on the evaluation dataset). AutoML however figures out on its own how to create these splits, so we won't need to do that here. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bq = bigquery.Client(project=PROJECT)\n",
    "title_dataset = bq.query(query).to_dataframe()\n",
    "title_dataset.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "AutoML for text classification requires that\n",
    "* the dataset be in csv form with \n",
    "* the first column being the texts to classify or a GCS path to the text \n",
    "* the last colum to be the text labels\n",
    "\n",
    "The dataset we pulled from BiqQuery satisfies these requirements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"The full dataset contains {n} titles\".format(n=len(title_dataset)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make sure we have roughly the same number of labels for each of our three labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "title_dataset.source.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally we will save our data, which is currently in-memory, to disk.\n",
    "\n",
    "We will create a csv file containing the full dataset and another containing only 1000 articles for development.\n",
    "\n",
    "**Note:** It may take a long time to train AutoML on the full dataset, so we recommend to use the sample dataset for the purpose of learning the tool. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATADIR = './data/'\n",
    "\n",
    "if not os.path.exists(DATADIR):\n",
    "    os.makedirs(DATADIR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "FULL_DATASET_NAME = 'titles_full.csv'\n",
    "FULL_DATASET_PATH = os.path.join(DATADIR, FULL_DATASET_NAME)\n",
    "\n",
    "# Let's shuffle the data before writing it to disk.\n",
    "title_dataset = title_dataset.sample(n=len(title_dataset))\n",
    "\n",
    "title_dataset.to_csv(\n",
    "    FULL_DATASET_PATH, header=False, index=False, encoding='utf-8')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's sample 1000 articles from the full dataset and make sure we have enough examples for each label in our sample dataset (see [here](https://cloud.google.com/natural-language/automl/docs/beginners-guide) for further details on how to prepare data for AutoML)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_title_dataset = title_dataset.sample(n=1000)\n",
    "sample_title_dataset.source.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's write the sample datatset to disk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SAMPLE_DATASET_NAME = 'titles_sample.csv'\n",
    "SAMPLE_DATASET_PATH = os.path.join(DATADIR, SAMPLE_DATASET_NAME)\n",
    "\n",
    "sample_title_dataset.to_csv(\n",
    "    SAMPLE_DATASET_PATH, header=False, index=False, encoding='utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_title_dataset.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import shutil\n",
    "\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.callbacks import TensorBoard, EarlyStopping\n",
    "from tensorflow.keras.layers import (\n",
    "    Embedding,\n",
    "    Flatten,\n",
    "    GRU,\n",
    "    Conv1D,\n",
    "    Lambda,\n",
    "    Dense,\n",
    ")\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "from tensorflow.keras.preprocessing.text import Tokenizer\n",
    "from tensorflow.keras.utils import to_categorical\n",
    "\n",
    "\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by specifying where the information about the trained models will be saved as well as where our dataset is located:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "LOGDIR = \"./text_models\"\n",
    "DATA_DIR = \"./data\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our dataset consists of titles of articles along with the label indicating from which source these articles have been taken from (GitHub, Tech-Crunch, or the New-York Times)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATASET_NAME = \"titles_full.csv\"\n",
    "TITLE_SAMPLE_PATH = os.path.join(DATA_DIR, DATASET_NAME)\n",
    "COLUMNS = ['title', 'source']\n",
    "\n",
    "titles_df = pd.read_csv(TITLE_SAMPLE_PATH, header=None, names=COLUMNS)\n",
    "titles_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Integerize the texts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first thing we need to do is to find how many words we have in our dataset (`VOCAB_SIZE`), how many titles we have (`DATASET_SIZE`), and what the maximum length of the titles we have (`MAX_LEN`) is. Keras offers the `Tokenizer` class in its `keras.preprocessing.text` module to help us with that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = Tokenizer()\n",
    "tokenizer.fit_on_texts(titles_df.title)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "integerized_titles = tokenizer.texts_to_sequences(titles_df.title)\n",
    "integerized_titles[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "VOCAB_SIZE = len(tokenizer.index_word)\n",
    "VOCAB_SIZE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATASET_SIZE = tokenizer.document_count\n",
    "DATASET_SIZE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_LEN = max(len(sequence) for sequence in integerized_titles)\n",
    "MAX_LEN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now implement a function `create_sequence` that will \n",
    "* take as input our titles as well as the maximum sentence length and \n",
    "* returns a list of the integers corresponding to our tokens padded to the sentence maximum length\n",
    "\n",
    "Keras has the helper functions `pad_sequence` for that on the top of the tokenizer methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 1\n",
    "def create_sequences(texts, max_len=MAX_LEN):\n",
    "    sequences = tokenizer.texts_to_sequences(texts)\n",
    "    padded_sequences = pad_sequences(sequences, max_len, padding='post')\n",
    "    return padded_sequences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sequences = create_sequences(titles_df.title[:3])\n",
    "sequences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "titles_df.source[:4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now need to write a function that \n",
    "* takes a title source and\n",
    "* returns the corresponding one-hot encoded vector\n",
    "\n",
    "Keras `to_categorical` is handy for that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "CLASSES = {\n",
    "    'github': 0,\n",
    "    'nytimes': 1,\n",
    "    'techcrunch': 2\n",
    "}\n",
    "N_CLASSES = len(CLASSES)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 2\n",
    "def encode_labels(sources):\n",
    "    classes = [CLASSES[source] for source in sources]\n",
    "    one_hots = to_categorical(classes)\n",
    "    return one_hots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "encode_labels(titles_df.source[:4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing the train/test splits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's split our data into train and test splits:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "N_TRAIN = int(DATASET_SIZE * 0.80)\n",
    "\n",
    "titles_train, sources_train = (\n",
    "    titles_df.title[:N_TRAIN], titles_df.source[:N_TRAIN])\n",
    "\n",
    "titles_valid, sources_valid = (\n",
    "    titles_df.title[N_TRAIN:], titles_df.source[N_TRAIN:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To be on the safe side, we verify that the train and test splits\n",
    "have roughly the same number of examples per classes.\n",
    "\n",
    "Since it is the case, accuracy will be a good metric to use to measure\n",
    "the performance of our models.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sources_train.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sources_valid.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using `create_sequence` and `encode_labels`, we can now prepare the\n",
    "training and validation data to feed our models.\n",
    "\n",
    "The features will be\n",
    "padded list of integers and the labels will be one-hot-encoded 3D vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, Y_train = create_sequences(titles_train), encode_labels(sources_train)\n",
    "X_valid, Y_valid = create_sequences(titles_valid), encode_labels(sources_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y_train[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building a DNN model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The build_dnn_model function below returns a compiled Keras model that implements a simple embedding layer transforming the word integers into dense vectors, followed by a Dense softmax layer that returns the probabilities for each class.\n",
    "\n",
    "\n",
    "Note that we need to put a custom Keras Lambda layer in between the Embedding layer and the Dense softmax layer to do an average of the word vectors returned by the embedding layer. This is the average that's fed to the dense softmax layer. By doing so, we create a model that is simple but that loses information about the word order, creating a model that sees sentences as \"bag-of-words\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_dnn_model(embed_dim):\n",
    "\n",
    "    model = Sequential([\n",
    "        Embedding(VOCAB_SIZE + 1, embed_dim, input_shape=[MAX_LEN]), # TODO 3\n",
    "        Lambda(lambda x: tf.reduce_mean(x, axis=1)), # TODO 4\n",
    "        Dense(N_CLASSES, activation='softmax') # TODO 5\n",
    "    ])\n",
    "\n",
    "    model.compile(\n",
    "        optimizer='adam',\n",
    "        loss='categorical_crossentropy',\n",
    "        metrics=['accuracy']\n",
    "    )\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we train the model on 100 epochs but adding an `EarlyStopping` callback that will stop the training as soon as the validation loss has not improved after a number of steps specified by `PATIENCE` . Note that we also give the `model.fit` method a Tensorboard callback so that we can later compare all the models using TensorBoard."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "tf.random.set_seed(33)\n",
    "\n",
    "MODEL_DIR = os.path.join(LOGDIR, 'dnn')\n",
    "shutil.rmtree(MODEL_DIR, ignore_errors=True)\n",
    "\n",
    "BATCH_SIZE = 300\n",
    "EPOCHS = 100\n",
    "EMBED_DIM = 10\n",
    "PATIENCE = 0\n",
    "\n",
    "dnn_model = build_dnn_model(embed_dim=EMBED_DIM)\n",
    "\n",
    "dnn_history = dnn_model.fit(\n",
    "    X_train, Y_train,\n",
    "    epochs=EPOCHS,\n",
    "    batch_size=BATCH_SIZE,\n",
    "    validation_data=(X_valid, Y_valid),\n",
    "    callbacks=[EarlyStopping(patience=PATIENCE), TensorBoard(MODEL_DIR)],\n",
    ")\n",
    "\n",
    "pd.DataFrame(dnn_history.history)[['loss', 'val_loss']].plot()\n",
    "pd.DataFrame(dnn_history.history)[['accuracy', 'val_accuracy']].plot()\n",
    "\n",
    "dnn_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building a RNN model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `build_dnn_model` function below returns a compiled Keras model that implements a simple RNN model with a single `GRU` layer, which now takes into account the word order in the sentence.\n",
    "\n",
    "The first and last layers are the same as for the simple DNN model.\n",
    "\n",
    "Note that we set `mask_zero=True` in the `Embedding` layer so that the padded words (represented by a zero) are ignored by this and the subsequent layers.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_rnn_model(embed_dim, units):\n",
    "\n",
    "    model = Sequential([\n",
    "        Embedding(VOCAB_SIZE + 1, embed_dim, input_shape=[MAX_LEN], mask_zero=True), # TODO 3\n",
    "        GRU(units), # TODO 5\n",
    "        Dense(N_CLASSES, activation='softmax')\n",
    "    ])\n",
    "\n",
    "    model.compile(\n",
    "        optimizer='adam',\n",
    "        loss='categorical_crossentropy',\n",
    "        metrics=['accuracy']\n",
    "    )\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's train the model with early stoping as above. \n",
    "\n",
    "Observe that we obtain the same type of accuracy as with the DNN model, but in less epochs (~3 v.s. ~20 epochs):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "tf.random.set_seed(33)\n",
    "\n",
    "MODEL_DIR = os.path.join(LOGDIR, 'rnn')\n",
    "shutil.rmtree(MODEL_DIR, ignore_errors=True)\n",
    "\n",
    "EPOCHS = 100\n",
    "BATCH_SIZE = 300\n",
    "EMBED_DIM = 10\n",
    "UNITS = 16\n",
    "PATIENCE = 0\n",
    "\n",
    "rnn_model = build_rnn_model(embed_dim=EMBED_DIM, units=UNITS)\n",
    "\n",
    "history = rnn_model.fit(\n",
    "    X_train, Y_train,\n",
    "    epochs=EPOCHS,\n",
    "    batch_size=BATCH_SIZE,\n",
    "    validation_data=(X_valid, Y_valid),\n",
    "    callbacks=[EarlyStopping(patience=PATIENCE), TensorBoard(MODEL_DIR)],\n",
    ")\n",
    "\n",
    "pd.DataFrame(history.history)[['loss', 'val_loss']].plot()\n",
    "pd.DataFrame(history.history)[['accuracy', 'val_accuracy']].plot()\n",
    "\n",
    "rnn_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build a CNN model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `build_dnn_model` function below returns a compiled Keras model that implements a simple CNN model with a single `Conv1D` layer, which now takes into account the word order in the sentence.\n",
    "\n",
    "The first and last layers are the same as for the simple DNN model, but we need to add a `Flatten` layer betwen the convolution and the softmax layer.\n",
    "\n",
    "Note that we set `mask_zero=True` in the `Embedding` layer so that the padded words (represented by a zero) are ignored by this and the subsequent layers.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_cnn_model(embed_dim, filters, ksize, strides):\n",
    "\n",
    "    model = Sequential([\n",
    "        Embedding(\n",
    "            VOCAB_SIZE + 1,\n",
    "            embed_dim,\n",
    "            input_shape=[MAX_LEN],\n",
    "            mask_zero=True), # TODO 3\n",
    "        Conv1D( # TODO 5\n",
    "            filters=filters,\n",
    "            kernel_size=ksize,\n",
    "            strides=strides,\n",
    "            activation='relu',\n",
    "        ),\n",
    "        Flatten(), # TODO 5\n",
    "        Dense(N_CLASSES, activation='softmax')\n",
    "    ])\n",
    "\n",
    "    model.compile(\n",
    "        optimizer='adam',\n",
    "        loss='categorical_crossentropy',\n",
    "        metrics=['accuracy']\n",
    "    )\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's train the model. \n",
    "\n",
    "Again we observe that we get the same kind of accuracy as with the DNN model but in many fewer steps.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "tf.random.set_seed(33)\n",
    "\n",
    "MODEL_DIR = os.path.join(LOGDIR, 'cnn')\n",
    "shutil.rmtree(MODEL_DIR, ignore_errors=True)\n",
    "\n",
    "EPOCHS = 100\n",
    "BATCH_SIZE = 300\n",
    "EMBED_DIM = 5\n",
    "FILTERS = 200\n",
    "STRIDES = 2\n",
    "KSIZE = 3\n",
    "PATIENCE = 0\n",
    "\n",
    "\n",
    "cnn_model = build_cnn_model(\n",
    "    embed_dim=EMBED_DIM,\n",
    "    filters=FILTERS,\n",
    "    strides=STRIDES,\n",
    "    ksize=KSIZE,\n",
    ")\n",
    "\n",
    "cnn_history = cnn_model.fit(\n",
    "    X_train, Y_train,\n",
    "    epochs=EPOCHS,\n",
    "    batch_size=BATCH_SIZE,\n",
    "    validation_data=(X_valid, Y_valid),\n",
    "    callbacks=[EarlyStopping(patience=PATIENCE), TensorBoard(MODEL_DIR)],\n",
    ")\n",
    "\n",
    "pd.DataFrame(cnn_history.history)[['loss', 'val_loss']].plot()\n",
    "pd.DataFrame(cnn_history.history)[['accuracy', 'val_accuracy']].plot()\n",
    "\n",
    "cnn_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright 2022 Google Inc. Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License"
   ]
  }
 ],
 "metadata": {
  "environment": {
   "name": "tf2-gpu.2-1.m68",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-1:m68"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
